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ABSTRACT 

This educational newsletter highlights a lead article, 
"Evidence-Based Research in Education." The article explains that evidence- 
based research emerged in the field of medicine over 50 years ago, resulting 
in major advances in the treatment and prevention of disease. It adds that 
clinical guidelines and protocols are based on the results of controlled 
experiments following rigorous standards of science. The U.S. Department of 
Education is embracing evidence-based research to improve the effectiveness 
of educational interventions and, in turn, academic achievement. The article 
discusses the No Child Left Behind Act and the Coalition for Evidence-Based 
Policy. It recommends strategies for the Department of Education to bring- 
evidence-driven progress to education. Other information materials in the 
newsletter are: "The What Works Clearinghouse"; "NDDR Grantees Review WWC 

Draft Standards"; and "Resolution of the AAMR on Evidence-Based Research and 
Intellectual Disability." (BT) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Evidence-based 
Research in Education 



Evidence-based research emerged in the field of medicine over 50 
years ago, and has resulted in major advances in the treatment and 
prevention of disease. Clinical guidelines and protocols are based on the 
results of controlled experiments following rigorous standards of science. 
Improvements in public health and reduced death rates have occurred not 
by chance but due to the requirements of the Federal Drug Administration 
(FDA) and National Institutes of Health (NIH) to implement effective, 
evidence-based research (Coalition for Evidence-Based Policy, 2002). 



The Cochrane Collaboration http://www.cochrane.org/ is an international 

__ — — — organization “that aims to help people make 
well informed decisions about health care 
by preparing, maintaining and ensuring 
the accessibility of systematic reviews of 
the effects of health care interventions” 
(Cochrane Collaboration, 2003). The first 
Cochrane Center was established in 1992 
to fulfill the legacy of Archie Cochrane, 
a British epidemiologist, who suggested 
the need for a systematic review of 
randomized controlled trials in order to 
understand health care and what was most 
— — — =» effective. The Cochrane Collaboration 

includes groups of researchers (and others) 
in many countries, that focus on reviewing studies in different health care 
areas and developing reliable reports to be archived in accessible databases. 



The U.S. Department of 
Education is embracing 
evidence-based research 
in order to improve the 
effectiveness of educational 
interventions and in turn, 
academic achievement. 



The U.S. Department of Education (ED) is embracing evidence-based 
research in order to improve the effectiveness of educational interventions 
and in turn, academic achievement. The “No Child Left Behind” Act 
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Evaluating the Quality 

of Research Findings 



Quality is an elusive characteristic that often derives 
as much from the “eye of the beholder 5 as it does 
from an objective application of a measurement 
standard. Additionally, quality often becomes defined 
in relation to what has happened, what has been the 
case, or what has been produced in the past. This is 
why statements associated with establishing criteria 
to distinguish quality from non-quality frequently 
change over time due to the relative and changing 
nature of our perception of what quality is. 

It is also the case that quality of information derived 
from research faces acceptance or rejection by others 
based on its perceived credibility. In the world of 
research, this is referred to as face validity. The 
perception of “high quality research results’ 5 can be 
influenced by a variety of factors ranging from the 
area selected for research and why, how the research 
was designed, diversity of the sample, data collection 
methods, partners in the research activity, data 
analysis methods, and statement of implications of the 
findings — just to name a few of the factors involved. 

In the real world, those learning of research findings 
usually are most concerned with: (a) Does it relate to 
me and a need I have today? and (b) Can I access it, 
use it, and benefit from it? 

NIDRRs Rehabilitation Research and Training 
Centers (NCDDR & NARRTC, 2003) used a 
consumer-oriented category system in summarizing 
their accomplishments. This system was based around 
quality of life questions a person with a disability 
often raises (Corrigan, 1994). These include: 



Today, the quality of research has been implied in a 
variety of national policies under a different name 
— scientifically based (or evidence-based) research. 

The National Research Council (2002) has suggested 
several principles to guide us in distinguishing 
scientifically based research from other forms of 
research. These principles describe scientifically based 
research as being: 

• driven by significant questions; 

• empirical in nature; 

• theory based; 

• designed around a sound linkage between 

the research question and the research 
method; 

• based on clear inferential reasoning; 

• capable of being replicated producing similar 

results; 

• generalizable to significant audiences; and 

• available for professional scrutiny. 

In linking quality considerations with information 
dissemination, a growing demand exists for 
information that has been determined to have reached 
an acceptable level of quality prior to making it 
available for utilization by others. This issue of The 
Research Exchange discusses one of these efforts being 
conducted by the What Works Clearinghouse. Their 
draft: criteria to judge quality of research results was 
shared with a group of NIDRR-sponsored researchers 
for their reactions and comments. These are 
included in this issue along with news about special 
recognitions and media featuring NIDRR grantees. 



• How long will I live? 

• What will I live on? 

• Where will I live? 

• What will I do? 

• Whom will I love? 

• How much choice will I have? 




h, yjurtu — 



John D. Westbrook, Ph.D. 
Director 



Most consumers, the ultimate beneficiaries of 
disability research efforts, would likely not find 
research to be of “high quality 55 unless it addressed at 

least one of these major quality of life questions. National Research Council. (2002). Scientific research in 

education. Washington, DC: National Academies Press. 




NCDDR & NARRTC. (2003). RRTC Highlights of 
Accomplishments. Austin: SEDL. Available: 
http://www.ncddr.org/du/products/rrtc-hilights/index.html 
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Evidence-based Research in Education 

continued from page 1 

(NCLB) signed by President Bush in 
2001 emphasizes accountability and the 
use of scientifically based research. 

The Coalition for Evidence-Based 
Policy has worked with ED since 2001 
to help define and forge a policy that 
supports evidence-based research. It 
sponsored a forum on November 18, 
2002, Rigorous Evidence: The Key to 
Progress in Education , that involved staff 
from ED, the White House, and the 
Office of Management and Budget, as 
well as representatives from education 
committees in the Congress, educators, 
advocates, and researchers. 

Bri ngi ng Evidence- Dr i ven Progress To 
Education: A Recommended Strategy for 
the U.S. Department of Education is a 
collaborative report by the Coalition 
that was released at the forum. The 
Executive Summary of the report offers 
the following suggestions: 

1. A major, Department-wide effort 
should be launched to: 

(i) Build the knowledge base 
of educational interventions 
proven effective through 
randomized controlled trials — 
not just in small demonstration 
projects but in large-scale 
replications; and 

(ii) Provide strong incentives 
for the widespread use of 
such proven interventions by 
recipients of federal education 
funds. 

2. In order to build the knowledge base 
of effective, replicable interventions 
in High Priority Areas: 

• The Department should focus its 
discretionary funds for research and 
evaluation on randomized trials to 
identify such interventions. 

• The Department’s grant programs 
should give applicants major 
incentives to use their discretionary 
funds to carry out such randomized 
trials. 

3. In order to provide strong incentives 
for the widespread use of proven 

O entions, the Departments 

ERIC - 



grant programs should require 
applicants to provide a plan for 
widespread implementation of 
research-proven interventions, with 
quantifiable goals. 

Coalition for Evidence- Based Policy. 

(2002.) Bringing Evidence-Driven Progress 
To Education: A Recommended Strategy 
for the U.S. Department of Education 
(Executive Summary.) Washington, 

DC: Author. Available: http:// 

www.excelgov.org/usermedia/images/ 

uploads/PDFs/CoalitionExSum.pdf 

The Education Sciences Reform Act 
(ESRA) of 2002 (P.L. 107-279) was 
signed on November 5, 2002. The 
purpose of the ESRA is “to provide 
for improvement of Federal education 
research, statistics, evaluation, 
information, and dissemination...” ESRA 
established the Institute of Education 
Sciences (IES) within ED, replacing 
the Office of Educational Research and 
Improvement (OERI). 

The IES reflects the intent of the 
President and Congress to advance the 
field of education through support of 
rigorous evidence-based educational 
research. Dr. Grover J. (Russ) Whitehurst 
was appointed as the first Director of 
the IES (http://www.ed.gov/about/ 
offices/list/ies/index.html). Whitehurst 
describes the current state of education 
as 90 percent professional wisdom based 
on experience, and 10 percent empirical 
evidence based on research. He wants to 
swap the percentages, using evidence- 
based or scientifically based research 
to guide educational improvement 
(Whitehurst, 2002. http://www.ed.gov/ 
nclb/ methods/whatworks/eb/edlite- 
slide021.html). 

The Individuals with Disabilities 
Education Act (IDEA) is due for 
reauthorization in 2003. Following 
the lead of the NCLB, Secretary of 
Education Rod Paige has emphasized 
the need for increased accountability 
and “doing what works” to improve 
educational outcomes for students with 
disabilities. 

"IDEA should ensure that schools, local 
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education agencies, state education 
agencies and the Federal Department 
of Education quickly adopt research 
and evidence-based practices. OSERS 
research and training activities should 
be aligned with the work of the 
Department’s Institute of Education 
Sciences. Additionally, information 
should be provided to families and 
teachers on effective programs based on 
rigorous research, including requiring 
the federally funded parent training 
centers to educate parents about 
effective research that improves results 
for students with disabilities. IDEA 
should also reflect the research principles 
outlined by the President’s Commission 
on Excellence in Special Education 
while adhering to the standards for 
high quality research established by the 
Education Sciences Reform Act of 2002" 
(http://www.ed.gov/news/pressreleases/ 
2003/02/02252003.html). 

The reauthorization of the Rehabilitation 
Act of 1973, as amended, is scheduled 
to occur prior to October 1, 2003. As 
this date draws near, it is likely that the 
changes seen in previous legislation 
may be discussed as part of the 
reauthorization process. 

Many NIDRR grantees are located 
in medical centers that are involved 
with evidence-based clinical research. 
However, many NIDRR grantees may 
not be conducting experimental research 
through randomized trials. 

Research that samples low-incidence 
students and adults with disabilities with 
unique conditions may not be feasible 
in research designs utilizing randomized 
trials. Groups such as the American 
Association for Mental Retardation have 
issued position statements describing 
the potential exclusion of people 
with developmental disabilities from 
“approved” evidence-based research 
(see related article, "Resolution of the 
AAMR on Evidence-Based Research and 
Intellectual Disability," on page 12). 
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The What Works Clearinghouse 



“The What Works 
Clearinghouse (WWC), 
launched in 2002, has 
specified clear and rigorous 
methodological standards 
for demonstrations of 
program effectiveness in 
education. ...We expect the 
Clearinghouse to become 
the principal source of valid 
information on effective 
educational practice” 
(Whitehurst, 2003). 

In his March 13, 2003 statement before 
the U.S. House Subcommittee on 
Labor/HHS/Education Appropriations, 
Assistant Secretary Whitehurst noted 
that the WWC was the focal point of 
the nearly $20 million requested for 
dissemination activities within the IES 
budget (Whitehurst, 2003. 
http://www.ed.gov/ news/speeches/ 
2003/03/03 132003a.html). 

The WWC was funded to identify 
educational interventions and programs 
that are based on research conducted 
with rigorous experimental methods. 
Whitehurst noted that in the No Child 
Left Behind Act (NCLB), the term 
scientifically based research is found 111 
times. The NCLB defines scientifically 
based research as “...research that 
involves the application of rigorous, 
systematic, and objective procedures 
to obtain reliable and valid knowledge 
relevant to education activities and 
programs” (Whitehurst, 2002. http:// 
www.ed.gov/ nclb/ methods/whatworks/ 
eb/ edlite-slideO 1 3 .html) . 

Providing evidence. 
Improving education. 

The WWC has been established by 
the U.S. Department of Educations 
. . 9 te of Education Sciences to 

ERIC 



provide educators, policymakers, and the 
public with a central, independent, and 
trusted source of scientific evidence of 
what works in education. The WWC is 
administered by ED through a contract 
to a joint venture of the American 
Institutes for Research and the Campbell 
Collaboration, internationally recognized 
leaders in the fields of education research 
and rigorous reviews of scientific 
evidence. 

The Cochrane Collaboration serves as a 
model for the Campbell Collaboration, 
which focuses on the “social, behavioral, 
and educational arenas” (Campbell 
Collaboration, 2003). Established in 
2000, the Campbell Collaboration works 
closely with the Cochrane group (see 
related article on page 5). 

Subcontractors for the WWC include 
Aspen Systems Corporation and 
Caliber Associates, Inc. Aspen provides 
technological, communications, and 
meeting support, including designing 
and maintaining the WWC Web 
site and databases, publishing an 
electronic news bulletin, and producing 
outreach products. Caliber Associates is 
contributing to the first year education 
topic reviews for the WWC. 

The WWC seeks input from the 
education community and encourages 
you to visit the Web site 
http://w-w-c.org/ for more information 
on the WWC and how you can be 
involved. To receive regular updates 
on the work of the WWC, you may 
send your name and email address to 
wwcinfo@w-w-c.org, or subscribe to 
WWCUpdate at: 
http://w-w-c.org/list.html 

Scope of Work 

The WWC scope of work includes 
support for informed education 
decision-making by producing WWC 
Evidence Reports, high-quality reviews 
of scientific evidence of the effectiveness 
of educational programs, practices, 
products, and policies. Through a set 



of accessible Web-based databases, the 
WWC will provide decision makers 
with the information they need to make 
choices based on high-quality scientific 
research. The WWC is to provide: 

• reviews of potentially replicable 
interventions (programs, products, 
and practices) that are intended to 
enhance student outcomes; 

• information about the evaluation 
studies on which intervention reviews 
have been based; 

• scientifically rigorous reviews of 
test instruments used to assess 
educational effectiveness; and 

• outcome evaluators (individuals 
and organizations) willing to 
conduct evaluations of educational 
interventions. 

The WWC databases are intended to 
complement those of the current ERIC 
Clearinghouses. The administration of 
the ERIC system is being recompeted 
and the current Clearinghouses are 
scheduled to begin phasing out in 
December 2003. 

WWC Standards 

The WWC s standards for reviewing 
studies and interventions for scientific 
evidence of educational effectiveness 
will be used to prepare WWC Evidence 
Reports on identified topic areas. 

These documents — the Study 
Design and Implementation Device 
( Study DIAD) and the Cumulative 
Research Evidence Assessment Device 
( CREAD) — will be used to review 
and report on the characteristics 
of individual educational- 
effectiveness studies, identify the 
strengths and weaknesses of each 
study’s methodology, and assess 
the strength of the conclusions 
that can be drawn from a body of 
research on specific educational 
interventions — that is, well-defined 
and replicable programs, products, 
practices, and policies. 
(http://www.w-w-c.org/ 
pr_062003 .html) 

5 ' 
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The draft Study DIAD \ used to review 
individual studies and interventions, 
was posted on the WWC Web site and 
online comments were accepted from 
November through December, 2002. In 
addition, a forum through which more 
public input was received, was held 
on November 22, 2002. Feedback, a 
summary document and report on the 
forum were posted on the WWC’s Web 
site in January 2003. An initial draft 
of the CREAD , which is designed for 
use in reviewing a cumulative body of 
work, was posted in January 2003 and 
comments were taken. 

Versions 1.0 of the Study DIAD and 
of the CREAD were approved by the 
WWC’s Technical Advisory Group 
(TAG) in June 2003. The Study DIAD , 
Version 1.0 was posted on the WWC 
Web site in July, and Version 1.0 of the 
CREAD will be posted in August 2003. 

First Year Topic Areas 

WWC Evidence Report topic areas 
are chosen to meet the needs of K-12, 
adult educators, and education decision 
makers to identify and implement 
effective approaches to improving 
student outcomes. Each year, the WWC 
is to develop an agenda for the education 
topic areas it will review. Nominations 
for future topic areas are continually 
accepted via the WWC Web site. 
Nominations should meet the definitions 
and criteria outlined at 
http://www.w-w-c.org/ topicnom.html 

The topic areas identified by the WWC 
for first year review are: 

• Interventions for Beginning Reading 

• Curriculum-Based Interventions for 
Increasing K-12 Math Achievement 

• Preventing High School Dropout 

• Programs for Increasing Adult 
Literacy 

• Peer-Assisted Learning in Elementary 
Schools: Reading, Mathematics, and 
Science Gains 

• Interventions To Reduce Delinquent, 
Disorderly, and Violent Behavior in 
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The Campbell 

The Campbell Collaboration is 
an international organization that 
aims to prepare, maintain, and 
disseminate high-quality, systematic 
reviews of studies of effectiveness 
of social and educational policies 
and practices. By supporting the 
production of these reviews and by 
disseminating results in an accessible 
fashion, the Campbell Collaboration 
intends to contribute to decisions 
in practice, policy and public 
understanding. 

The Campbell Collaboration 
collaborates with its sibling 
organization, the Cochrane 
Collaboration, which prepares and 
maintains systematic reviews of the 
effects of interventions in health 
care. Established in 2000, the 
Campbell Collaboration is named 
after an American psychologist 
and thinker, Donald Campbell, 
who drew attention to the need for 
societies to assess more rigorously the 
effects of their social and educational 
experiments, that is, the policies and 
practices that they introduce and 
promote. 

The Campbell Education Group 
aspires to help teachers, parents, 
students, school administrators, and 
education policy makers make well- 



informed decisions about teaching 
and learning by putting the best 
available evidence from systemic 
reviews of educational research at 

the heart of educational policies and 

3 

practice. 

The Campbell Collaboration - 
American Institutes for Research 
Joint Venture was established 
specifically to develop and maintain 
the What Works Clearinghouse, and 
brings together nationally recognized 
leaders in the field of rigorous 
reviews of scientific evidence. 

The chair of the Steering Group 
of the Campbell Collaboration, 
Robert Boruch, serves as Principal 
Investigator for the WWC. 



Sources: 

1 Campbell Collaboration Methods Group 
Guiding Principles 

http://www.missouri.edu/^c2method/ 

2 The Campbell Collaboration - 
A Brief Introduction 
http://www.aic.gov.au/campbellcj/ 
intro.html 

3 Campbell Collaboration 
Coordinating Groups 
http://www.campbellcollaboration.org/ 
cccgroup.html 

4 U.S. Department of Education 
Awards Contract for ‘What 
Works Clearinghouse’ 

http:/ /www. ed.gov/ news/pressreleases/ 
2002/08/08072002a.htmi 



Middle and High Schools 

• Interventions for Elementary 
School English Language Learners: 
Increasing English Language 
Acquisition and Academic 
Achievement. 

The WWC Web site information does 
not specifically mention, nor at this 
time, include specific interventions 
for students with disabilities. In the 
future, the WWC hopes it will produce 



Evidence Reports that specifically focus 
on interventions targeting students 

with disabilities. At present, whenever 

possible, if there are high-quality 

studies that break out findings by 

subcategories the WWC will report 

findings on students with disabilities 

within all relevant Evidence Reports. It 

should be noted that the WWC does 

not conduct field research, and therefore 

depends on existing research to produce 

its Evidence Reports. 
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NIDDR Grantees Review 
WWC Draft Standards 



Although much NIDRR-funded 
research is focused on areas outside of 
K-12 education, the NCDDR believes 
it is important to make all NIDRR 
grantees aware of the evidence-based and 
scientifically based emphasis that ED is 
placing on research and interventions 
for education. For projects that do focus 
on K-12, it is critical for their NIDRR- 
funded research to be included in the 
WWC databases. The NCDDR has 
begun an activity to gather information 
developed by NIDDR grantees that 
addresses infants, children and youth 
with disabilities. This activity is being 
conducted in order to bring forward 
NIDRR-sponsored research that may 
appropriately be included in the WWC 
research collection. 

NCDDR staff asked researchers from 
Rehabilitation Research and Training 
Centers (RRTCs) and a sample of other 
NIDRR grantees to review the draft 
documents Study DIAD , Version 0.6 and 
CREAD , Version 0.6, along with other 
information available from the WWC 
Web site. 

Study DIAD 

The Study Design and Implementation 
Assessment Device ( Study DIAD) “is 
a system for assessing the degree to 
which the design and implementation 
of individual evaluations permit 
conclusions about the causal effects of an 
intervention” (p.l, Valentine & Cooper, 
2003). The designers of the Study DIAD 
“attempted to create one instrument 
that can be used to answer questions 
at four different levels of specificity. 

We wanted the most general level to 
be understandable to an audience of 
nonresearchers and the most detailed 
level to be specific enough to satisfy 
researchers’ desire for comprehensiveness 
and explicitness. 







The questions are arranged so that 
answers at one level — the most specific 
level — feed into a set of design and 
implementation questions at a second 
level, a set of Composite Questions at a 
third level, and then into a fourth level 
of even more general questions” (p. 2-3, 
Valentine & Cooper, 2003). 

Valentine, J. C., & Cooper, H. 

(2003). What Works Clearinghouse 
Study Design and Implementation 
Assessment Device (Version 1.0). 
Washington, DC: U.S. Department of 
Education 

NIDRR grantees reviewed the draft 
Version 0.6 (posted 3/11/03) 
http : / / w- w-c. o r g/ s tandards 06.doc 

Approved in June 2003, Version 1.0 is 
now available (posted 7/17/03) 
http://wwwc.org/DIAD_Final.doc 

CREAD 

Companion to the Study DIAD is the 
Cumulative Research Evidence Assessment 
Device , (CREAD) Version 1.0. The 
CREAD is a device that is meant to 
“provide an expression of the confidence 
with which a conclusion can be drawn 
about the existence of causal effects of an 
intervention based on an entire body of 
accumulated evidence.” ...“The CREAD 
will assess confidence in causal inferences 
based upon the (a) number, (b) validity 
characteristics, (c) variations, and 
(d) consistency in results of studies 
included in the evidence base” 

(p. 1, Cooper & Valentine, 2003). 

The CREAD and the Study DIAD both 
utilize Cook & Campbell’s “threats to 
validity” framework. The CREAD uses 
three dimensions that are unique to 
evaluating the certainty of inferences 
permitted by a set of studies, or an 
evidence base: depth, breadth, and 
consistency (p. 2, Cooper & Valentine, 
2003). 



Cooper, H. & Valentine, J.C. 
(2003). What Works Clearinghouse 
Cumulative Research Evidence Assessment 
Device (Version 0.6). Washington, DC: 
US Department of Education. 

NIDRR grantees reviewed draft Version 
0.6 (posted 1/23/03) http://w-w-c.org/ 
creadv06.doc 

Approved in June 2003, Version 1.0 
will be posted on the WWC Web site in 
August 2003. 



Comments on Study DIAD 
and CREAD 

Selected NIDRR researcher were asked 
to submit any questions about how 
the standards and review process might 
apply to NIDRR-funded research, 
specifically: 

(a) Overarching questions regarding 
how “quality of research results” 
will be established through these 
standards, and 

(b) Questions that may be relevant to 
establishing the extent to which 
research results that may be included 
in the WWC’s databases would 
adequately address the needs of 
individuals with disabilities. 

Comments and questions about the 
Study DIAD and CREAD , versions 0.6, 
were received from 14 NIDRR grantees, 
and they were forwarded to staff of the 
WWC for their comment. 



Respondent 1: 

of Currently, the Study DIAD only 

covers papers reporting intervention 
studies, and within that category 
only those that were designed as 
intervention studies. Other designs 
that often are used to evaluate 

7 
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interventions (any retrospective 
designs, e.g. case control) are 
not covered. Certain prospective 
intervention designs are also not 
covered, (e.g. single subject designs, 
case series and other “weak” 
designs). Many of these approaches 
to research yield useful data that 
would either be not counted or 
discounted by the Study DLAD. 

cP NIDRR grantees doing intervention 
research often produce multiple 
publications, all relevant to the 
main research question or major 
research hypothesis. Study DIAD 
has no mechanism to handle this. 
(However, if the publications do not 
“contradict” one another - different 
number of cases, etc. - the outcomes 
could be combined for a single 
Study DIAD scoring). 

cP Based on their research data 

NIDRR grantees often also publish 
papers that address side questions, 
ad hoc hypotheses, etc., for which 
the research design that was used 
to answer the main question may 
not be optimal. Study DIAD 
should not be used to condemn 
papers addressing “peripheral” 
issues and ad-hoc questions; as 
pilot/exploratory work they may 
help the science move forward, and 
should be published with credit 
given to NIDRR for facilitating 
“innovation.” 

cP Study DIAD does not come up 

with one total score/judgment, but 
with four; possible answers include 

“YES,” “NO,” “MAYBE YES” 
and “MAYBE NO.” These latter 
two judgments are not sufficiently 
grounded to serve as a basis for 
grant program management 
decision making. Similarly, there is 
no mechanism to combine just YES 
and NO judgments into an overall 
judgment of a paper, let alone a 
grant-funded research program. 

A While criteria for what is good and 
what is poor research practice (see 
p. 5 of Study DIAD) are pretty 
"" and agreed upon among 




researchers and writers of research 
methodology textbooks, there is less 
consensus as to what is good or best 
or acceptable practice in a resource- 
poor environment: limited funding, 
overly restrictive human subjects 
requirements that result in large 
and selective attrition, and other 
logistical problems that are outside 
the investigators control. (Especially 
as uncontrolled/unplanned events 
unfold after a study has been 
started, e.g. the introduction of 
HIPAA). Study DIAD does not 
allow a judgment of “adequate given 
the circumstances.” 

Bringing forward the science of 
effective interventions is a good thing; 
the judgment on the usefulness and 
effect size of an intervention should be 
based strictly on the results of a study 
as implemented. For management of 
a research program this is a bad thing; 
judgments on productivity should take 
into account the fact that resources may 
be limited, or that after the start of a 
research study there were changes in the 
environment of the study that made 
implementation of the original plan 
difficult if not impossible. 

Possible refinements to Study DIAD: 

□P Include questions addressing 

whether the experimental or the 
control groups may have been 
exposed to interventions outside 
the study (co-interventions) that 
may have enhanced or diminished 
the effect of the experimental 
intervention. Similarly missing from 
the Study DIAD are other aspects 
of history, maturation, testing, 
instrumentation, etc. that imperil 
internal validity. 

Following are specific comments about 
the eight Composite Questions in the 
Study DIAD. These will be identified as 
CQl (Composite Question HI), etc. 

cP Construct validity-intervention 
addressed in CQl and construct 
validity-outcome measures in CQ2 
do not seem to reflect traditional 
issues of construct validity (which 



refer to the adequacy of the data 
produced by an instrument in 
quantifying a construct), but to 
other issues such as: 

— Was the intervention 

properly defined, described, 
implemented? (issues of 
treatment fidelity) 

— Was the outcome measure 
relevant to the intervention? 
(only CQ2.2 addresses 
construct validity proper) 

cP It is possible to have a misguided, 
muddled, undefined, undescribed 
intervention that is measured 
(exquisitely, but wrongly) using a 
psychometrically strong instrument. 
The same is possible for outcomes: 
the instrument is a great one, but it 
has just no relevance to the concept 
the researcher aims to measure. 

The correspondence between 
concept and measurement data is 
the domain of construct validity; to 
use the same term for other issues 
just confuses us even more than we 
already are with the 213 types of 
“validity” researchers have invented 
to date. 

cP CQ3 dealing with internal 

validity-selection (for randomized 
experiments, quasi-experimental, 
and regression discontinuity 
designs) does not address adequacy 
of the randomization scheme and 
of the randomization concealment. 
These have been shown to be 
major problems in clinical medical 
research. 

a 0 CQ5 combines questions on 
sampling of cases, sampling of 
outcomes and sampling of post- 
intervention time points. The latter 
two issues might be better placed 
in CQ2 that deals with outcome 
measures. 

cP Issues of negative side effects, 
balance of benefits and risks are 
not included. Are these irrelevant 
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to study quality, and should they 
be picked up elsewhere by the 
evidence report team? Or is a 
measure of poor study quality the 
fact that side/negative effects are 
not mentioned? 

□° CQ6.3 in Composite Question #6 

addresses testing across outcomes. 
This would also better be placed 
in CQ2 dealing with outcome 
measures. 

CQ6.4 asks about testing for 
time of measurement; needs 
clarification by rephrasing. 

CQ6.5 on testing intervention 
variations might be better placed 
in CQl (construct validity- 
interventions). 

cP CQ7 addresses statistical validity- 
effect size estimation; however, the 
valid use of statistical tests is not 
judged. 

CQ7.3 and CQ7.4 address 
sample size and outcome 
measurement error. These roll 
up into the precision of effect 
sizes, for example, the confidence 
interval. Effect sizes can be 
calculated whatever sample size 
and measurement error; whether 
the estimate of effect size is 
“sufficiently precise” is fairly 
irrelevant. If the various studies 
surviving Study DIAD review are 
combined, a joint study effect 
size will be calculated, with its 
own precision estimate. That’s the 
time to worry about “sufficiently 
precise.” 

cP CQ8 addresses the adequacy 
of reporting of statistical 
tests, NOT whether the tests 
performed themselves were 
adequate or appropriate (e.g. that 
parametric tests were not used 
for non-parametric data). In this 
Composite Question , Q8.3a is a 
relevant question, if effect sizes 
could be estimated for important 
measured outcomes. If the answer 
rn Q8.3a is YES, the answer 



to Q8.1 (if sample sizes were 
reported/estimable) is YES, and 
the answer to Q8.2 (if directions 
of effects could be identified) 
is YES. If the answer to Q8.3a 
is NO, questions Q8.1 , Q8.2 , 
and CQ8.3b wold be irrelevant. 
Either you can calculate an effect 
size (using standard procedures 
or advanced mathematics), or 
you cannot. If you cannot, the 
study cannot contribute to the 
Evidence Report - unless studies 
summarized in “effect: yes, effect 
size: unknown” are included. 

cP If some of the items in CQ7 
and CQ8 are deleted, it may be 
useful to combine the remaining 
questions into one Composite 
Question. 

CREAD 

The Cumulative Research Evidence 
Assessment Device {CREAD) suggests 
that Study DIAD Composite Question 
#8 is based on a judgment of 
censoring/publication bias. The Study 
DIAD does not mention this problem; 
in fact, a judgment of publication 
bias cannot be made based on a single 
study, and that’s likely why the Study 
DIAD is silent as to this problem. 



Respondent 2: 

I appreciate the work that the panel 
put into the project. I would like 
to emphasize that some of my most 
serious concerns are not related to 
the documents per se, but rather with 
what actions the developers can take to 
ensure that the information is used in 
the most appropriate ways. 

First, I would like to applaud the 
overall quality of the effort. The 
transparency of the process and 
algorithms for including, excluding, 
and aggregating the studies is a very 
positive step. I appreciate the efforts 
of the group to be maximally scientific 
and dispassionate in applying the 
criteria to the various studies. This is 
particularly true of the Study DIAD, 
the CREAD , and then the actual 



reports as written, will obviously 
contain relatively greater elements 
of subjectivity. While this is not 
necessarily a bad thing, I think that 
it is important to acknowledge that 
there is no failsafe road to objective 
recommendations. Since most people 
will not be aware of the limitations of 
the method involved in reaching the 
final report, I think it is important 
to include relevant disclaimers and 
cautions, and remind people that 
this is an interpretation of available 
information. I do not think this is a 
fault with the method of the WWC, 
rather it is inherent in this sort of 
effort. 

I do have several concerns with the 
particulars of the Study DIAD/CREAD , 
however, which I list below. I will 
include first those most directly related 
to the nature of disability studies, 
and follow with comments related to 
studies in education more generally. 

My primary concern from the 
perspective of someone who has a 
focus on children with disabilities 
stems from the restrictions regarding 
the design of studies that are to be 
considered for review under the 
guidelines for the Study DIAD. The 
forward to the document notes that 
the review will target studies that 
are capable of adducing causality; 
however, randomized trials and quasi- 
experimental methods are certainly not 
the only means for doing so. 

Researching interventions for children 
with emotional and behavioral 
disorders (EBD) and/or developmental 
disabilities (DD), much well-regarded 
research rests on within-subjects 
designs which, when well done, 
certainly can assess causal effectiveness 
and deserve to be called “scientific.” 

By omitting these types of designs, 
the universe of “scientific” studies 
leaves out much of what we know to 
be effective in educational settings 
with children with EBD and DD. For 
example, there are many studies on 
positive behavioral supports, effective 
teacher interventions to increase 
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academic engagement, peer-mediated 
learning and social skills strategies. 

In a similar vein, though of less 
particular concern to the population 
we tend to study, is that other sorts of 
designs are also left out, (e.g. regression 
discontinuity - although I suppose this 
can be considered a form of quasi- 
experimental design, it doesn’t appear 
to fit from the criteria as currently 
listed). Methodologically speaking, I 
believe inferences regarding causality 
drawn from regression discontinuity 
designs are on a par with randomized 
experiments. 

My other concerns relate to 
interventions in education more 
generally, and are thus relevant both to 
disabled (EBD/DD) and nondisabled 
populations. I will use examples from 
research on reading interventions, since 
this is a hotly contested issue currently 
and also this is one of the first topics 
that the WWC is intending to address. 

First, I appreciate the concern of the 
developers of the Study DIAD with the 
alignment of outcomes, particularly 
of the over alignment of outcomes 
with interventions — for example 
this will address in a productive way 
the issue of whether teaching kids 
phonics by reading pseudowords and 
then measuring intervention group 
versus control on reading pseudowords 
actually has anything to say about 
decoding of real words, much less 
comprehension. In other studies, there 
is even a trickier distinction to make 
(e.g. is fluency accurately measured by 
read-aloud outcomes only?). This can 
only be a judgment call, and I think 
that consumers of the WWC reports 
should be made aware of how the 
reporters have come down on this and 
other similarly controversial issues. 

Even with this attention to alignment 
of immediate outcomes, I have serious 
concerns about long-term outcomes 
from reading interventions as well as 
the effect over time that might result 
from people consulting the WWC 
or similar sources. There is a sort of 

o 
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dynamic over time that can occur, 
and I think we have seen evidence 
for some of this already in the area 
of reading: Short-term studies with 
easy-to-measure outcomes are the 
easiest to undertake. Many are done, 
these approaches appear promising 
(and may be recommended as “what 
works”, which is legitimate) and are 
funded more frequently. Textbook 
companies structure materials around 
these short-term approaches (since 
political pressures push for short-term, 
easy-to-measure outcomes), and these 
approaches acquire the status of dogma 
even though there is little evidence of 
positive long-term outcomes that are 
meaningful in terms of preparing kids 
for advanced study and careers. 

Recent international comparisons 
show, for example, that U.S. kids 
receive more hours per week in reading 
instruction than international peers. 
And while the average reading is on 
par with international peers (at fourth 
grade measures), an astonishingly 
high percentage of U.S. fourth 
graders NEVER read for fun outside 
of school (32 percent vs. 18 percent 
internationally). Longer term outcomes 
for reading levels of US high school 
students compare less favorably to 
international outcomes, particularly 
for certain groups of kids (who happen 
to be also those least likely to read for 
fun). I think it is certainly a reasonable 
hypothesis that increasing demand 
to focus on drill-type, skill-building, 
direct instruction methods for 
reading can have the perverse effect of 
increasing short-term outcomes while 
having a long-term negative impact on 
kids’ desire to read and their ultimate 
academic success. 

In the results from the National 
Reading Panel, we saw this sort of 
dynamic more or less in effect, with 
the result that there has been increasing 
pressure on schools to adopt a very 
limited number of prepackaged reading 
programs. And while the studies cited 
in the NRP report found the largest 
effect for relatively small amounts 
of instruction in some of these basic 

to 



skills (e.g. phonemic awareness and 
phonics) the pressure in classrooms has 
been to do more and more, with very 
little attention to comprehension or 
stimulation of interest in reading. 

I realize that these concerns are 
somewhat outside the immediate 
pervue of the Study DIAD/CREAD 
developers, and I do not mean to 
imply that it is a shortcoming of the 
approach; however , I also think that 
the developers have a responsibility to 
anticipate how this sort of information 
will be used, and to package and 
contextualize it, and to provide explicit 
commentary of what the results do 
and do not allow us to say. Hopefully, 
this will minimize the potential for 
misinterpretation or misuse of the 
information. Thank you for the 
opportunity to provide input into this 
interesting and useful project. 



Respondent 3: 

Overall, I found the standards to 
be clear and comprehensive as they 
pertained to research that lends itself 
to a clean experimental or quasi- 
experimental design. The value of 
the guidelines is less clear in areas of 
disability support and disability policy 
where experimental designs are more 
difficult to implement, or, because of 
the dispersed nature of the population 
and the difficulty of exerting 
experimental controls in community 
environments, establishing adequate 
power and comparable populations 
will be more difficult to achieve. These 
issues are particularly true for low 
incidence populations. 

A second area of concern is the role of 
the growing emphasis on experimental 
design in relation to the broad mission 
of NIDRR around disability policy 
and the structure and organization of 
disability and generic supports. This 
concern is particularly evident for a 
Center like ours that focuses on state 
service systems. Our broad mission to 
describe and improve the delivery of 
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employment supports. Our work is at 
a state structure and state policy level, 
and interventions necessarily involve a 
complex set of variables that don’t lend 
themselves well to experimental control. 
For example, the total n, from a state 
perspective, is 50 (or 51 with DC). 

An intervention in one state is very 
unlikely to be directly comparable to an 
intervention in another state given the 
variation in political and cultural factors, 
lack of experimental controls over the 
implementation process, and a host of 
other variables. 

Some general questions are listed below. 
Thanks for the opportunity to comment. 

□ D To what extent will the WWC 
guidelines be applied to 
nonacademic areas including 
transition planning, employment 
supports, and development of self- 
determination skills? 

cP How will effective practices for low 
incidence populations be addressed 
in the implementation of the 
guidelines? 

a 0 How will the guidelines address 

larger issues like self-determination, 
employment support, and related 
transition needs for students from 
low incidence populations? It will 
be difficult to amass sufficient 
evidence for many of these 
populations. 

a 0 How will qualitative research 
be incorporated in the overall 
guidance that the WWC provides 
to practitioners? In particular, 
qualitative research may serve 
two roles. As a precursor to the 
review process it serves to frame 
hypotheses and the parameters of 
interventions to allow for effective 
design of experimental studies. 
Second, qualitative research may 
provide closer description of the 
intervention, and help practitioners 
to identify the key parameters 
and relevant variations in the 
intervention. It seems that the 




WWC should provide for review 
and incorporation of qualitative 
research as part of its process. 

cP How will the WWC address 

complex systems-level change such 
as the design and implementation 
of employment supports within 
state government? 

cP How might single subject 
experimental designs be 
incorporated in the analysis? 

Respondent 4: 

cP I am enthusiastic about the 

adoption of Campbell’s framework 
regarding threats to validity. I’ve 
used Campbell &C Stanley, and 
Cook &C Campbell, for many years 
to prepare young researchers to 
evaluate their (and others’) research 
designs. The approach is eminently 
reasonable for application to the 
breadth of research funded by 
NIDRR, OSEP, IES and other 
entities. I think it serves us well. 

□ D The hierarchical arrangement of 
questions in both tools is also 
reasonable and, furthermore, I 
found the wording of the questions 
to be comprehensible and suitable. 

cP In terms of questions, I admit to an 
inadequate understanding of some 
key aspects of the presentation. For 
instance, 

- It is not clear to me how the 
tools would be used, by whom, 
at what point, and for what 
particular purposes. A number of 
the questions call for judgments, 
so the issue of who the reviewers 
would be is vital. 

- Some parts of the operations 
(procedures) are confusing to 
me. For example, I admit (with 
hesitation) that I do not grasp the 
statement on the tables, “Read 
down to determine the answer to 
the question.” 

- In the CREAD , what determines 
the set of studies that would be 
included in the ,eyluation of the 



consistency, depth and breadth of 
the evidence base? 



Respondent 5: 

Before I make specific comments on 
these documents, I pose a couple of 
caveats based upon my own research and 
that of my agency. 

(1) Although much of our research 
concerns “intervention,” none of 
our research concerns “educational” 
interventions — which is the specific 
research focus of these devices. 

(2) As stated in the introductory 
remarks, these devices do not address 
qualitative research (the primary 
type of research I do) nor other types 
of research that is not randomized 
nor focused on uncovering causal 
relationships. 

As such, it is difficult to for me to 
evaluate how well these assessments 
apply to the types of research we are 
involved in here. I did find parts of the 
introductory explanations to be lacking 
sufficient clarity. 

I’m concerned how well either of these 
evaluation devices work in practice. 

That is, I’d like to see evidence that 
these evaluation devices actually reflect 
how the evaluator(s) would otherwise 
assess the research. Have these evaluation 
devices been piloted with people 
outside the group that developed them? 
Following are some specific concerns 
about the Study DIAD: 

cP On page 4: The fourth assumption 
discussed is that Donald Campbell’s 
approach is “an obvious and natural 
choice.” I’m not familiar with 
his work, so it doesn’t seem either 
obvious or natural to me. It’s not 
that I’d reject his approach, but I’d 
certainly want more discussion or 
description to inspire a little more 
confidence. 

cP On page 6: If the authors of a study 
did not report how participants 
were allocated to groups, that 
study would be given a no’ on the 
randomized assessment question. 
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Why “no” instead of “not available?” 
From this example, I’m concerned 
that the overall push to reduce 
the research to fit an algorithmic 
flow chart under- or over-estimates 
confidence in these evaluations. 

□P CQ6 ; Was the intervention tested for 
its effect within important subgroups 
of participants, settings , outcomes, 
occasions, and intervention variations ? 
What does "important" mean here? 

0 ° On page 7: I appreciate the attempt 
to be transparent and flexible 
regarding definition of terms. 
However, if each time “vague 
or ambiguous” terms are given 
meaning by different leaders doing 
the evidence reports, then how will 
possible disagreements or differences 
be accounted for when trying to 
do cumulative evaluations under 

CREAD? 

Concern about the CREAD , page 4: 

The authors raise the question of how 
inconsistency of findings will be treated, 
and suggest that some issues “will be 
resolved uniformly” while “other issues 
may be left: up to the evidence report 
team” which “will have to exercise good 
professional judgement in making these 
determinations.” While I appreciate the 
complexity of this issue, it leaves me 
uncertain how consistent and objective 
these cumulative evaluations would be. 

I think questions of consistency across 
projects are particularly noteworthy 
when considering much of NIDRR s 
work across different disabilities. That is, 
how will possibly similar interventions 
with different disability populations be 
addressed? 

Respondent 6: 

This is an excellent project. I have some 
comments regarding the two documents: 

Social science research design 
and analysis is also tied to ethical 
considerations. That is, all else equal, 
research with better methodological 
design is of higher ethical quality. Unless, 
I missed this point, much more could 
t O >out the quality of the research 
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endeavor and ethical considerations. For 
instance, with external validity issues, 
meaningful involvement of persons 
with disabilities in designing and 
interpreting disability policy research can 
be important. Regarding the CREAD , 

I have argued that meta-analysis, where 
possible, is an ethical imperative. 

Respondent 7: 

I read over the WWC’s CREAD and the 
Study DIAD, and I think that they are 
both excellent. Clearly, a lot of work 
has gone into these documents. My 
question would be, are they going to be 
doing something similar for single study 
subject research designs? 

These documents appear to orient 
towards research that is looking at 
interventions done at the classroom 
level. A lot of rehabilitation, say, 
rehabilitation for kids with Cerebral 
Palsy who might have impaired speech 
or language processing, or kids with 
other physiological disorders who might 
have impaired cognitive processing, is 
a tailored program where the person 
serves as his or her own control. Because 
there is no way that there would ever 
be enough kids, except in the very, very 
largest districts, to get a large group and 
then randomize them to interventions. 



Respondent 8: 

My primary concern is that all 
evidence-based practice carry with it 
documentation of representativeness 
(external validity) of population. 
Specifically, for example, if an 
intervention or practice is considered 
evidence -based and recommended or 
used with American Indians, American 
Indians should be documented as having 
been included in the research sample. 



Respondent 9: 

The documents look very good. They 
reflect the best in the construction of 
group designs. The lack of discussion 
of single subject design is of concern, 
however. First-rate group design can be 
very difficult to do with low incidence 
populations, where sample sizes are 
12 



small and populations may be pretty 
heterogeneous. For these populations, 
single subject design can be the best way 
to demonstrate treatment effects. 



Comments from 
Researchers on 
Application of the WWC 
Standards to IMSDRR 
Research 

Several NIDRR grantee respondents 
expressed concerns that the Study 
DIAD and CREAD could be applied to 
NIDRR-funded research for purposes 
other than consideration for the WWC 
databases. The following comments 
reflect some of those perceptions: 



Respondent 1 : 

In general our biggest issue was how 
these documents would be used by 
NIDRR. As a tool to evaluate “centers 
of excellence,” investigators, or research, 
more work needs to be done. 

a 0 Study DIAD cannot be used 
effectively prior to study 
implementation and reporting. The 
quality and quantity of a study’s 
implementation (e.g. treatment 
fidelity, number of subjects) affect 
Study DIAD scores, as does the 
quality of reporting of the study 
results. Thus, it is no use when 
applied to the evaluation of research 
proposals, i.e., peer review. 

0 ° NIDRR grantees doing intervention 
research often produce multiple 
publications, all relevant to the 
main research question/major 
research hypothesis. Study DIAD 
has no mechanism to handle this. 
(However, if the publications do not 
“contradict” one another - different 
number of cases, etc. - the outcomes 
could be combined for a single 
Study DIAD scoring). 

NIDRR Grantee Review WWC Draft Standards 
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cP Based on their research data 

NIDRR grantees often also publish 
papers that address side questions, 
ad hoc hypotheses, etc., for which 
the research design that was used 
to answer the main question may 
not be optimal. Study DIAD should 
not be used to condemn these 
papers addressing “peripheral” issues 
and ad-hoc questions: as pilot or 
exploratory work they may help 
the science move forward, and 
should be published with credit 
given to NIDRR for facilitating 
“innovation.” 



Respondent 3: 

Establishing true experiments on a scale 
adequate for meeting the guidelines 
will require a substantial change in the 
allocation of funds to research activities. 
What should the role of NIDRR be, and 
what are the implications for funding 
levels usually associated with FIP and 
RRTC activities? Might NIDRR serve as 
a R&D arm of ED, framing issues that 
need a larger scale' experimental analysis? 



Respondent 5: 

Overall, I do appreciate the attempt to 
standardize how research design and 
implementation are assessed. However, I 
definitely have questions about how this 
type of evaluation would be applied to 
NIDRR research (i.e., by ignoring other 
types of research and/or implying that 
other types of research are less desirable 
if they are not oriented to causal effect- 
randomized samples). 

If NIDRR were to consider using these 
evaluation tools for (at least some of) 
its research, I think NIDRR needs to 
articulate what the purpose would be (as 
described in the Study DIAD). 



Respondent 6: 

What is the intended use of these 
instruments, as relates to the 
functioning and evaluation of NIDRR s 
Rehabilitation Research and Training 
Centers? 

Is the idea to apply this model that is 
designed for educational research to the 
clinical, policy, and socio-environmental 
research conducted by NIDRR RRTC 
grantees? 



n 0 Would the model be applied as is, 
or modified? 

n 0 If to be modified, who would 
be involved in the modification 
process? 

n 0 If to be modified, would one new 
tool with modifications be applied 
to all RRTCs, or would specific 
modifications be made to cohorts of 
RRTCs with similar functions and 
focuses? 

cP Should these tools be applied, who 
would be the reviewers that would 
use the tools to review RRTC 
functioning? 

□° Would results of the review be used 
for program evaluation purposes? 

cP Who would use these results 

(NIDRR staff, external reviewers, 
or the program staff themselves 
for internal program evaluation 
purposes)? 



Resolution of the AAMR cm Evidence-Based 

Research and Intellectual Disability 



The move toward evidence-based research as specified 
in the No Child Left Behind legislation is wise, but its 
overly restrictive definition harms people with intellectual 
disabilities and those with other severe disabilities. Any 
definition of evidence-based research must include 
methodologies which are widely regarded as scientifically 
defensible (such as single subject experimental design). 

The requirement that research methods be restricted to group 
design with a preference for randomized clinical trials will 
significantly inhibit the development and validation of new 
scientific knowledge in education. There are many situations 
in which it is not feasible to assign children at random 
to school placements or to types of instructional settings. 

Yet these situations do not preclude other viable research 
designs. These concerns are especially great for people with 
intellectual and developmental disabilities given their low 
incidence and their unique characteristics which require 
individualization of educational intervention. 



We strongly urge Congress, the U.S. Department of 
Education, and other federal agencies to recognize and 
support the continued use of the full array of research 
methodologies that are scientifically accepted in the fields of 
education, psychology, and child development. We request 
that a committee of highly respected educational researchers 
in low incidence disabilities be convened without delay 
to develop appropriate language on the array of scientific 
research methods for incorporation into the reauthorization 
of IDEA. 

January 13, 2003 
AAMR Board of Directors 

American Association on Mental Retardation Board of 
Directors. (2003). Resolution of the AAMR on Evidence-Based Research 
and Intellectual Disability. Retrieved August 1, 2003 from 
http://www.aamr.org/Reading_Room/pdf/resolution_research.pdf 

Copyright 2003 AAMR. Reprinted with permission. 
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Online Resources 



Response from the 
WWC to NIDRR 
Researchers’ Comments 

Full responses from the WWC were not 
received in time for this issue, but the 
NCDDR will share them in a separate 
document at a later date. Following are 
the WWC s preliminary comments: 

The WWC thanks the NIDRR grantees 
for your thoughtful comments on 
the Study Design and Implementation 
Assessment Device (Study DIAD )> Version 
0.6 and the Cumulative Research Evidence 
Assessment Device ( CREAD ) Version 0.6. 
We do want to note that the Study DIAD 
and CREAD were posted for public 
comment (November-December 2002 
and January-February 2003 respectively) 
and that the opportunity for public 
comment was widely circulated among 
education constituencies. The Feedback 
on Draft Standards can be viewed at 
http:/ / w-w-c.org/ standards.html 

The FAQ’s posted at http://w-w-c.org/ 
faqs/index.html provide additional 
context and clarification about the 
standards and other aspects of the 
WWC’s work. As you will see if you 
review our Web site at http://w-w-c.org/, 
many of the comments and questions 
you have raised are addressed in these 
documents. 

Public comments, as well as those offered 
by the WWC’s Technical Advisory Group 
(TAG), were incorporated in the revised 
standards. Version 1.0 of both the Study 
DIAD and the CREAD were recently 
approved by the TAG. The WWC 
will revisit, and if necessary revise and 
improve upon the standards periodically, 
based on comments such as the ones you 
have offered and experiences in using the 
standards in Evidence Reports. 

The WWC looks forward to providing 
responses to your questions and 
comments in the near future. We 
welcome your further comments or 
questions on the WWC. Please address 
them to: wwcinfo@w-w-c.org 
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Coalition for Evidence-Based Policy 

http : / / www. excelgov. o rg/ displayConte 
nt.asp?Keyword=prppcHomePage 



• Rigorous Evidence: The Key to 
Progress in Education 

h t tp : / / www. excelgov. o rg/ 

displayContent.asp?Keyword= 

prppcEvidence 

Education Sciences Reform Act (ESRA) 

http : / / www. ed . gov/ p olicy/ rschs tat/leg/ 
PL107-279.pdf 

Evidence Based Education (EBE) 
(Whitehurst, 2002). 

http://www.ed.gov/ nclb/ methods/ 
whatworks/ eb/ edlite-index.html 

Institute for Education Sciences (IES) 

http://www.ed.gov/ about/ offices/list/ 
ies/index.html 

No Child Left Behind Act (NCLB) 

http:/ /www. ed.gov/policy/ elsec/leg/ 
esea02/index.html 

NCLB Web site 

http://www.nclb.gov 

Reforming and Reauthorizing IDEA. 

H.R. 1350, Improving Results for 
Children With Disabilities Act of 2003 
http://edworkforce.house.gov/issues/ 

1 0 8 th/ educatio n/idea/idea. h tm 

Reauthorization of the Rehabilitation 
Act (ILRU/TIRR) 

http : / / www. ilru.org/ ilnet/ rehab/ 

Resolution of the AAMR on Evidence- 
Based Research and Intellectual 
Disability 

http:/ /www. aamr.org/Reading_Room/ 
pdf/resolution_research.pdf 

* t 



RSA - Legislation & Policy 

http://www.ed.gov/ about/ offices/list/ 
osers/ rsa / policy.html 

Statement before the House 
Subcommittee on Labor/HHS/ 
Education Appropriations, 

(Whitehurst, 2003). 
http://www.ed.gov/news/speeches/ 
2003/03/03 132003a.html 

What Works Clearinghouse 

http:/ /w- w-c.org/ 

• American Institutes for 
Research (AIR) 

http:/ /www. air.org/ 

• Aspen Systems Corporation 

http://www.aspensys.com/ 

• Caliber Associates, Inc. 

http://www.calib.com/home/ 
index. cfm 

• Campbell Collaboration 

http :/ / www. Campbell 
collaboration.org/ 

EVIDENCE-BASED MEDICINE 

• Agency for Health Care Research 
and Policy (AHRQ) 

http://www.ahrq.gov/ 

• Centre for Evidence 
Based Medicine 

http://www.cebm.net/ 

• Centre for Health Evidence - 
Users’ Guides to Evidence-Based 
Practice 

http://www.cche.net/ usersguides/ 
main. asp 

• Cochrane Collaboration 

http://www.cochrane.org/ 
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• Bringing Evidence-Driven Progress 
To Education: A Recommended 
Strategy for the U.S. Department of 
Education (Executive Summary.) 

http :/ / www. excelgov. org/ 
usermedia/ images/ uploads/ 

PDFs/ CoalitionExSum.pdf 
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Members of the NCDDR 
staff are on the lookout 
for popular and disability 
media pieces that present 
research funded by 
NIDRR. In this issue, we 
share items from: 

• Disability Network 
newsletter 

• The Boston Globe , 
EContent Magazine, and 
Government Computer 
News 



• Daily Mason Gazette, 
George Mason University 



Who*s in the News is a 
feature of The Research 
Exchange that shares 
some of the stories about 
NIDRR grantees and their 
research that have appeared in national 
media sources. Staff members will talk 
with grantees and media representatives 
about the origin and evolution of the 
stories, and their interactions with 
media representatives. Sharing this 
may be helpful to other grantees who 
would like to establish relationships 
with journalists and work with them to 
make information about their research 
available to the public. 



Please let the NCDDR know when 
an item representing your NIDRR- 
funded project appears in the media. 
Call us, 1-800-266-1832, or send 
email to ncddr@sedl.org and the 
item will be reviewed for Whos in 
the News. You may also use an online 
form: http://www.ncddr.org/forms/ 
submitnews.html 




The announcement of the 
IT Works Ability Awards was 
highlighted in Diversity 
World’s Disability Network 
newsletter, February 7, 2003. The 
Awards were established by the 
Information Technology Association of 
America (ITAA), in cooperation with 
the University of Iowa Law, Health 
Policy & Disability Centers NIDRR- 
funded IT Works project. Read the 
^ -y World newsletter article: 

ERIC 



http://www.diversityworld.com/ 

Disability/newsletter.htm#February 

%207,%202003 

IT Works Ability Awards is a national 
awards program “to stimulate interest in 
employing individuals with disabilities 
and to give public recognition and 
reward to IT firms that have developed 
effective strategies that promote the 
employment and advancement of 
people with disabilities.” 

(ITAA, 2003, http://www.itaa.org/ 
workforce/gendoc.cfm?DocID= 101) 
Press release from University of Iowa: 
http://www.uiowa.edu/^ournews/ 
2002/ may/ 0509disability-center.html 

The 2003 award winners were 
announced on May 5 at the ITAA’s 
sixth annual National IT Workforce 
Convocation in Arlington, Virginia 
and included IBM (Recruitment and 
Hiring), Microsoft (Accommodations), 
and Xerox (Developing Accessible IT 
Products and Services). 

Press Release about the Awards 
announcement on the ITAA Web site: 
http://www.itaa.org/news/pr/Press 
Release. cfm?ReleaseID= 1052825625 

For more information, 
contact IT Works Project director, 
James Schmeling: 
james-schmeling@uiowa.edu 

The Boston Globe (Business 
section), EContent 
Magazine , and Government 
Computer News picked 
up a press release about the new 
Accessible Technology Knowledgebase 
(ATKB) implemented by the NIDRR- 
funded Information Technology 
Technical Assistance & Training 
Center (ITTATC). Based in Atlanta at 
the Georgia Institute of Technology, 
ITTATC has added a powerful and 
user-friendly natural language search 
capability to their Web site. 

Searches will automatically query the 
Web sites and databases of the following 
federally funded disabilities-related 
resources. ^ 
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• AccessIT 

http://www.washington.edu/accessit/ 

• AssistiveTech.net 
http://www.assistivetech.net 

• DisabilityInfo.gov 
http://www.disabilityinfo.gov 

• ITTATC.org 
http://www.ittatc.org 

• Section508.gov 
http://www.section508.gov 

Partners in the ATKB project include 
iPhrase Technologies, Inc.; IDEAL 
Group, Inc.; NCR Corporation; and 
ITTATC. iPhrase is completing the 
ATKB s year-end advanced tuning 
process. Advanced tuning helps to 
insure the results of a search produce 
knowledge that is as current and 
relevant as possible. 

For more information, contact Mimi 
Kessler, ITTATC Project Director 
mimi.kessler@ittatc.org 

• Boston Globe. February 24, 2003 
by Peter J. Howe iPhrase behind 
government initiative to help disabled 
http://www.iphrase.com/PDFs/ 
02-24-03_BostonGlobe.pdf 

• EContent Magazine. 

February 25, 2003 
http://www.econtentmag.com/ 
Articles/ArticlePrint.aspx?ArticleID 
=4078&CategoryID=18 

• Government Computer News. April 
1, 2003 http://www.gcn.com/voll_ 
no 1 / daily-updates/ 215 44- 1 .html 

• iPhrase Press Release, 

February 24, 2003 http:// 
iphrase.com/news/feb2403.html 

• About the ITTATC Accessible 
Technology Knowledgebase (ATKB) 
http://www.ittatc.org/search_tips.cfm 

An article by Robin Herron 
appeared in the Daily Mason 
Gazette , George Mason 
University, on March 11, 2003 
describing the Middle School 
Phonemic Awareness Study. The study 
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is a field-initiated project funded by 
NIDRR and directed by Dr. Barbara 
Given, Associate Professor, Graduate 
School of Education at George Mason 
University. Dr. Given is director of 
adolescent and adult learning research 
projects at the Krasnow Institute for 
Advanced Study at GMU. 

The project focuses on the role of 
phonemic awareness in adolescents 
with low reading skills. Herrons 



article describes the study and how 
it is intended to help students make 
reading gains at a middle school in 
the Fairfax County Public Schools in 
Alexandria, Virginia. Glasgow Middle 
School, site of the Middle School 
Phonemic Awareness Study, was also 
named an “Accelerative Learning Model 
School” by the International Alliance 
for Learning, for its part in Dr. Givens 
research efforts. 



For more information about the 
study, contact Dr. Barbara Given: 
bgiven@gmu.edu 

“Reading Research Program Benefits 
Fairfax County Middle School 
Students” 

http:/ / gazette.gmu.edu/ articles/ 

index.php?id=4385 



R 
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The NCDDR 
continues to share 
the recognition given 
to NIDRR-funded 
researchers and their 
staff. The items 
presented in The 
Research Exchange 
demonstrate the wide 
variety and prestige of 
special awards made 
to staff members 
of NIDRR-funded 
projects across the 
country. 



All grantees are 
encouraged to send 
this information to the 
NCDDR for future 
issues. Send email to ncddr@sedl.org, 
call 1-800-266-1832, or use the online 
form available on the NCDDR Web 
site: http://www.ncddr.org/forms/ 
submitrecog.html 




Judy Brewer, Director of the 
Web Accessibility Initiative 
(WAI) at the World Wide 
Web Consortium (W3C), 
was honored by the Alliance 
for Public Technology (APT) with the 
Susan G. Hadden Pioneer Award. She 
was recognized “for pioneering efforts 
in telecommunications and consumer 
access” at a luncheon held February 
21, 2003 at the National Press Club in 
Washington, DC. 



The Susan G. Hadden Pioneer Award 
honors the memory of APT s Public 
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Policy Committee Chair who was killed 
while a tourist in Cambodia, in January 
1995. APT leaders called Dr. Hadden, 
who was Professor of Public Affairs at 
the LBJ School of Public Affairs at the 
University of Texas in Austin, “unique 
in her mastery of the complex tangle 
of regulatory and technical issues that 
accompany the current debate over 
telecommunications.” The award 
recognizes those who continue Haddens 
legacy of ensuring equitable access to 
technology as a democratizing principle. 

For more information, contact 

Judy Brewer: jbrewer@w3.org 







Media Access Generator 
software, MAGpie 2.01, 
recently received Honorable 
Mention for Best Educational 
Streaming Program in Streaming 
Magazines 2003 Readers Choice 
awards. Developers of Web- and 
CD-ROM-based multimedia need 
an authoring tool for making their 
materials accessible to persons with 
disabilities. The CPB/WGBH National 
Center for Accessible Media (NCAM) 
has developed MAGpie version 1.0 and 
2.01 for creating captions and audio 
descriptions for rich media. Funding 
was received from NIDRR, the Trace 
Research and Development Center, 
and the Mitsubishi Electric America 
Foundation. 



For more information about MAGpie 
2.01, contact: ncam_tech@wgbh.org 
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jJSSSk Dr. Judith Cook has received 
{UjjljW the American Sociological 
Associations (ASA) 2003 
William Foote Whyte 
Distinguished Career Award. Dr. 

Cook is the Principal Investigator of 
the NIDRR-funded UIC National 
Research and Training Center on 
Psychiatric Disability, University of 
Illinois at Chicago. 

Dr. Cook was honored for “her success 
in applied research, her commitment 
to participatory action research (PAR), 
and her effective work as a mentor to a 
large number of students.” The William 
Foote Whyte Distinguished Career Award 
is presented by the Sociological Practice 
Section of the ASA to an individual 
who has made notable contributions to 
sociological practice which can include 
several of the following elements: 
outstanding clinical or applied work, 
exceptional service to the section, 
publications that advance both the 
theory and methods of sociological 
practice, or mentoring and training 
of students for careers in sociological 
practice. The award honors the ASA’s 
72nd President. 

For more information, contact Ms. Edie 
Bamberger: bamberg@psych.uic.edu 




In October, 2002, Mitchell 
Rosenthal, Ph.D., Project 
Director of the Traumatic 
Brain Injury National 



NIDRR Grantee and Staff Recognition 

continued on page 16 
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NIDRR Grantee and Staff Recognition 

continued from page 1 5 

Data Center, was awarded the 2002 
Gold Key Award of Merit from the 
American Congress of Rehabilitation 
Medicine (ACRM), “in recognition 
of extraordinary service to the cause 
of rehabilitation medicine.” It is the 
highest honor given by ACRM. 

Dr. Rosenthal was also honored 
recently as the Leonard Diller Lecturer 
by the Division of Rehabilitation 
Psychology of the American 
Psychological Association at its 
conference “Rehabilitation Psychology 
2003” held in Tucson in April, 2003. 
Dr. Rosenthal is Vice-President 
for Research at Kessler Medical 
Rehabilitation Research and Education 
Corporation and Professor of Physical 
Medicine and Rehabilitation at the 
University of Medicine and 
Dentistry of New Jersey - New Jersey 
Medical School. 



For more information, contact Mitchell 
Rosenthal: mrosenthal@kmrrec.org 







A number of NIDRR grantees 
were named in the 14 th annual 
US News and World Report’s 
Best Hospitals. Seven of the 
top 17 hospitals on the 2003 
Honor Roll have NIDRR grants. These 
medical center and hospitals were highly 
ranked in 6 or more of the 17 Best 
Hospitals specialty areas, out of 203 
hospitals ranked. 



1. Johns Hopkins Hospital, Baltimore 

2. Mayo Clinic, Rochester, MN 

6. Duke University Medical Center, 
Durham, NC 



9. University of Michigan Medical 
Center, Ann Arbor 



10. University of Washington Medical 
Center, Seattle 

13. Stanford Hospital and Clinics, 
Stanford, CA 

17. Vanderbilt University Medical 
Center, Nashville 



See original article: 

http:/ / www.usnews.com/ usnews/ nycu/ 
health/hosptl/honorroll.htm 
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A total of 18 of the top 23 
Rehabilitation Hospitals are current 
grantees of NIDRR. The hospitals 
ranked in this list were identified by at 
least three percent of the board-certified 
physicians who responded to US, News 
surveys in 2001, 2002, and 2003. 

1 . Rehabilitation Institute of Chicago 

2. TIRR - The Institute for 
Rehabilitation and Research, 
Houston 

3. University of Washington Medical 
Center, Seattle 

4. Mayo Clinic, Rochester, MN 

5. Craig Hospital, Englewood, CO 

6. Kessler Institute for Rehabilitation, 
West Orange, NJ 

8. Thomas Jefferson University 
Hospital, Philadelphia 

9. Spaulding Rehabilitation Hospital, 
Boston 

10. Ohio State University Medical 
Center, Columbus 

1 1. Rancho Los Amigos National 
Rehabilitation Center, 

Downey, CA 

12. Johns Hopkins Hospital, Baltimore 

13. National Rehabilitation Hospital, 
Washington, DC 

14. University of Michigan Medical 
Center, Ann Arbor 

13. Moss Rehabilitation Hospital, 
Albert Einstein Medical Center, 
Philadelphia 

1 6. Shepherd Center, Atlanta 

17. Mount Sinai Medical Center, 

New York City 

18. Stanford Hospital and Clinics, 
Stanford, CA 

23. University of Alabama Hospital at 
Birmingham 

See original article: 
http://www.usnews.com/usnews/ 
nyc u/heal th/ hos p tl/rankings/ 
specrepreha.htm 

See full report: US News and World 
Report's Best Hospitals 2003 
http://www.usnews.com/ usnews/nycu/ 
health/hosptl/tophosp.htm 
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